User interface presenting an animated avatar performing a media reaction

ABSTRACT

This document describes techniques and apparatuses enabling a user interface for presenting a media reaction. The techniques receive media reactions of a person to a media program, such as the person laughing at one point of a comedy show, then smiling at another point, and then departing at a third point. The techniques may present these and other media reactions in a user interface through which a user may interact.

BACKGROUND

People often want to know what other people think about televisionshows, music, movies, and the like. Currently, a person may find out bycontacting his friends and asking them if they liked a particular movie,for example. This approach, however, can be time consuming, incomplete,or inaccurate. Asking friends may take a long while, some friends maynot have seen the movie, or some friends may have forgotten much oftheir impression of the program, and so reply with an inaccurateaccount.

A person may instead search out reviews of a program, such as publishedcritical reviews, or a source that averages online ratings from criticsor typical consumers. This approach, however, can also be time consumingor fail to help the person find out if he or she would like the programbecause the person may not have similar tastes to those of the moviecritic or typical consumer.

SUMMARY

This document describes techniques and apparatuses enabling a userinterface for presenting a media reaction. The techniques receive mediareactions of a person to a media program, such as the person laughing atone point of a comedy show, then smiling at another point, and thendeparting at a third point. The techniques may present these and othermedia reactions in a user interface through which a user may interact.In one embodiment, for example, the techniques present a time-basedgraph showing a person's reactions over the course of a media programand enabling selection to view the media reaction and/or a portion ofthe media program corresponding to the media reaction.

This summary is provided to introduce simplified concepts enabling auser interface presenting a media reaction, which is further describedbelow in the Detailed Description. This summary is not intended toidentify essential features of the claimed subject matter, nor is itintended for use in determining the scope of the claimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of techniques and apparatuses enabling a user interface forpresenting a media reaction are described with reference to thefollowing drawings. The same numbers are used throughout the drawings toreference like features and components:

FIG. 1 illustrates an example environment in which techniques enabling auser interface for presenting a media reaction can be implemented.

FIG. 2 is an illustration of an example computing device that is localto the audience of FIG. 1.

FIG. 3 is an illustration of an example remote computing device that isremote to the audience of FIG. 1.

FIG. 4 illustrates example methods for determining media reactions basedon passive sensor data.

FIG. 5 illustrates a time-based graph of media reactions, the mediareactions being interest levels for one person and for forty timeperiods during presentation of a media program.

FIG. 6 illustrates example methods for presenting a media reaction to amedia program.

FIG. 7 illustrates a tablet computer of FIG. 2 with a graphical userinterface presenting time-based graphs and ratings for three persons fora media program.

FIG. 8 illustrates example methods for presenting a media reaction to amedia program.

FIG. 9 illustrates an example device in which techniques enabling a userinterface presenting a media reaction can be implemented.

DETAILED DESCRIPTION Overview

This document describes techniques and apparatuses enabling a userinterface for presenting a media reaction. The techniques enable usersto know how other people reacted to various media programs. A user canquickly and easily see not only who viewed a media program, but whatportions they liked or did not like.

Consider, for example, a 30-minute situational comedy, such as TheOffice, which is typically 22 minutes in total content (eight of the 30minutes are advertisements). Assume that a user named Melody Pond hasnot yet watched a particular episode of The Office and wants to know howher sister Amelia Pond, her brother Calvin Pond, and her friend LydiaBrown liked the episode. The techniques enable a user interface thatpresents media reactions of all three people, assuming that all threehave watched that episode. This user interface can present overallimpressions of the episode, such as a rating (e.g., four stars), howengaged or interested each person was at each portion of the episode,and states (e.g., laughing, smiling, and talking) for each personthroughout the program.

Assume here that Amelia laughed and smiled through most of the episodebut cried at one point, Calvin smiled a couple times but was mostlydisinterested, and that Lydia was distracted through some portions butlaughed in many others. On seeing these media reactions, Melody mayselect to watch the portion of the episode that caused Amelia to cry,for example. On seeing this portion, Melody may better understand hersister or at least be able to talk with her about that portion of theepisode. Melody may instead decide not to watch the episode based on herknowing that she and Calvin have similar tastes in comedies becauseCalvin didn't seem to enjoy it. Or Melody may select to watch theepisode and have Amelia's reactions accompany the episode, such as withan avatar representing Amelia laughing and smiling during the episode atpoints where Amelia also laughed and smiled.

This is but one example of how techniques and/or apparatuses enabling auser interface presenting a media reaction can be performed. Techniquesand/or apparatuses that present and/or determine media reactions arereferred to herein separately or in conjunction as the “techniques” aspermitted by the context. This document now turns to an exampleenvironment in which the techniques can be embodied, after which variousexample methods for performing the techniques are described.

Example Environment

FIG. 1 is an illustration of an example environment 100 in which thetechniques enable a user interface for presenting a media reaction; thisenvironment is also one in which the techniques may receive sensor dataand determine media reactions based on this sensor data. Environment 100includes a media presentation device 102, an audience-sensing device104, a state module 106, an interest module 108, an interface module110, and a user interface 112.

Media presentation device 102 presents a media program to an audience114 having one or more persons 116. A media program can include, aloneor in combination, a television show, a movie, a music video, a videoclip, an advertisement, a blog, a web page, an e-book, a computer game,a song, a tweet, or other audio and/or video media. Audience 114 caninclude one or more multiple persons 116 that are in locations enablingconsumption of a media program presented by media presentation device102 and measurement by audience-sensing device 104, whether separatelyor within one audience 114. In audience 114 three persons are shown:116-1, 116-2, and 116-3.

Audience-sensing device 104 is capable of passively sensing audience 114and providing sensor data for audience 114 to state module 106 and/orinterest module 108 (sensor data 118 shown provided via an arrow). Inthis context, sensor data is passive by not requiring activeparticipation of persons in the measurement of those persons. Examplesof active sensor data include data recorded by persons in an audience,such as with hand-written logs, and data sensed from users throughbiometric sensors worn by persons in the audience. Passive sensor datacan include data sensed using emitted light or other signals sent byaudience-sensing device 104, such as with an infrared sensor bouncingemitted infrared light off of persons or the audience space (e.g., acouch, walls, etc.) and sensing the light that returns. Examples ofpassive sensor data and ways in which it is measured are provided ingreater detail below.

Audience-sensing device 104 may or may not process sensor data prior toproviding it to state module 106 and/or interest module 108. Thus,sensor data may be or include raw data or processed data, such as: RGB(Red, Green, Blue) frames; infrared data frames; depth data; heart rate;respiration rate; a person's head orientation or movement (e.g.,coordinates in three dimensions, x, y, z, and three angles, pitch, tilt,and yaw); facial (e.g., eyes, nose, and mouth) orientation, movement, orocclusion; skeleton's orientation, movement, or occlusion; audio, whichmay include information indicating orientation sufficient to determinefrom which person the audio originated or directly indicating whichperson, or what words were said, if any; thermal readings sufficient todetermine or indicating presence and locations of one of persons 116;and distance from the audience-sensing device 104 or media presentationdevice 102. In some cases audience-sensing device 104 includes infraredsensors (webcams, Kinect cameras), stereo microphones or directed audiomicrophones, and a thermal reader (in addition to infrared sensors),though other sensing apparatuses may also or instead be used.

State module 106 receives sensor data and determines, based on thesensor data, states of persons 116 in audience 114 (shown at arrow 120).States include, for example: sad, talking, disgusted, afraid, smiling,scowling, placid, surprised, angry, laughing, screaming, clapping,waving, cheering, looking away, looking toward, leaning away, leaningtoward, asleep, or departed, to name just a few.

The talking state can be a general state indicating that a person istalking, though it may also include subcategories based on the contentof the speech, such as talking about the media program (related talking)or talking that is unrelated to the media program (unrelated talking)State module 106 can determine which talking category through speechrecognition.

State module 106 may also or instead determine, based on sensor data, anumber of persons, a person's identity and/or demographic data (shown at122), or engagement (shown at 124) during presentation. Identityindicates a unique identity for one of persons 116 in audience 114, suchas Susan Brown. Demographic data classifies one of persons 116, such as5 feet, 4 inches tall, young child, and male or female. Engagementindicates whether a person is likely to be paying attention to the mediaprogram, such as based on that person's presence or head orientation.Engagement, in some cases, can be determined by state module 106 withlower-resolution or less-processed sensor data compared to that used todetermine states. Even so, engagement can be useful in measuring anaudience, whether on its own or to determine a person's interest usinginterest module 108.

Interest module 108 determines, based on sensor data 118 and/or aperson's engagement or state (shown with dashed-line arrow 126) andinformation about the media program (shown at media type arrow 128),that person's interest level (arrow 130) in the media program. Interestmodule 108 may determine, for example, that multiple laughing states fora media program intended to be a serious drama indicate a low level ofinterest and conversely, that for a media program intended to be acomedy, that multiple laughing states indicate a high level of interest.

As illustrated in FIG. 1, state module 106 and/or interest module 108provide demographics/identity 122 as well as one or more of thefollowing media reactions: engagements, shown at arrow 124, states,shown at arrow 120, or interest levels, shown at arrow 130. Based on oneor more of these media reactions, state module 106 and/or interestmodule 108 may also provide another type of media reaction, that ofoverall media reactions to a media program, such as a rating (e.g.,thumbs up or three stars). In some cases, however, media reactions arereceived and overall media reactions are determined instead by interfacemodule 110.

State module 106 and interest module 108 can be local to audience 114,and thus media presentation device 102 and audience-sensing device 104,though this is not required. An example embodiment where state module106 and interest module 108 are local to audience 114 is shown in FIG.2. In some cases, however, state module 106 and/or interest module 108are remote from audience 114, which is illustrated in FIG. 3.

Interface module 110 receives media reactions (overall or otherwise) anddemographics/identity information, and determines or receives someindication as to which media program the reactions pertain. Interfacemodule 110 presents, or causes to be presented, a media reaction (shownat arrow 132) to a media program through user interface 112. This mediareaction can be any of the above-mentioned reactions, some of which arepresented in a time-based graph to show a person's reaction over thecourse of the associated media program.

Interface module 110 can be local to audience 114, such as in caseswhere one person is viewing his or her own media reactions or those of afamily member. In many cases, however, interface module 110 receivesmedia reactions from a remote source.

FIG. 2 is an illustration of an example computing device 202 that islocal to audience 114. Computing device 202 includes or has access tomedia presentation device 102, audience-sensing device 104, one or moreprocessors 204, and computer-readable storage media (“media”) 206. Media206 includes an operating system 208, state module 106, interest module108, media program(s) 210, each of which may include or have associatedprogram information 212, interface module 110, and user interface 112.Note that in this illustrated example, media presentation device 102,audience-sensing device 104, state module 106, interest module 108, andinterface module 110 are included within a single computing device, suchas a desktop computer having a display, forward-facing camera,microphones, audio output, and the like. Each of these entities 102,104, 106, 108, and 110, however, may be separate from or integral witheach other in one or multiple computing devices or otherwise. As will bedescribed in part below, media presentation device 102 can be integralwith audience-sensing device 104 but be separate from state module 106,interest module 108, or interface module 110. Further, each of modules106, 108, and 110 may operate on separate devices or be combined in onedevice.

As shown in FIG. 2, computing device(s) 202 can each be one or acombination of various devices, here illustrated with six examples: alaptop computer 202-1, a tablet computer 202-2, a smart phone 202-3, aset-top box 202-4, a desktop 202-5, and a gaming system 202-6, thoughother computing devices and systems, such as televisions with computingcapabilities, netbooks, and cellular phones, may also be used. Note thatthree of these computing devices 202 include media presentation device102 and audience-sensing device 104 (laptop computer 202-1, tabletcomputer 202-2, smart phone 202-3). One device excludes but is incommunication with media presentation device 102 and audience-sensingdevice 104 (desktop 202-5). Two others exclude media presentation device102 and may or may not include audience-sensing device 104, such as incases where audience-sensing device 104 is included within mediapresentation device 102 (set-top box 202-4 and gaming system 202-6).

FIG. 3 is an illustration of an example remote computing device 302 thatis remote to audience 114. FIG. 3 also illustrates a communicationsnetwork 304 through which remote computing device 302 communicates withaudience-sensing device 104 (not shown, but embodied within, or incommunication with, computing device 202) and/or interface module 110.Communication network 304 may be the Internet, a local-area network, awide-area network, a wireless network, a USB hub, a computer bus,another mobile communications network, or a combination of these.

Remote computing device 302 includes one or more processors 306 andremote computer-readable storage media (“remote media”) 308. Remotemedia 308 includes state module 106, interest module 108, and mediaprogram(s) 210, each of which may include or have associated programinformation 212. Note that in this illustrated example, mediapresentation device 102 and audience-sensing device 104 are physicallyseparate from state module 106 and interest module 108, with the firsttwo local to an audience viewing a media program and the second twooperating remotely. Thus, as will be described in greater detail below,sensor data is passed from audience-sensing device 104 to one or both ofstate module 106 or interest module 108, which can be communicatedlocally (FIG. 2) or remotely (FIG. 3). Further, after determination bystate module 106 and/or interest module 108, various media reactions andother information can be communicated to the same or other computingdevices 202 for receipt by interface module 110. Thus, in some cases afirst of communication devices 202 may measure sensor data, communicatethat sensor data to remote device 302, after which remote device 302communicates media reactions to another of computing devices 202, allthrough network 304.

These and other capabilities, as well as ways in which entities of FIGS.1-3 act and interact, are set forth in greater detail below. Theseentities may be further divided, combined, and so on. The environment100 of FIG. 1 and the detailed illustrations of FIGS. 2 and 3 illustratesome of many possible environments capable of employing the describedtechniques.

Example Methods

FIG. 4 depicts methods 400 determines media reactions based on passivesensor data. These and other methods described herein are shown as setsof blocks that specify operations performed but are not necessarilylimited to the order shown for performing the operations by therespective blocks. In portions of the following discussion reference maybe made to environment 100 of FIG. 1 and entities detailed in FIGS. 2-3,reference to which is made for example only. The techniques are notlimited to performance by one entity or multiple entities operating onone device.

Block 402 senses or receives sensor data for an audience or person, thesensor data passively sensed during presentation of a media program tothe audience or person.

Consider, for example, a case where an audience includes three persons116, persons 116-1, 116-2, and 116-3 all of FIG. 1. Assume that mediapresentation device 102 is an LCD display having speakers and throughwhich the media program is rendered and that the display is incommunication with set-top box 202-4 of FIG. 2. Here audience-sensingdevice 104 is a Kinect, forward-facing high-resolution infraredred-green-blue sensor and two microphones capable of sensing sound andlocation that is integral with set-top box 202-4 or media presentationdevice 102. Assume also that the media program 210 being presented is aPG-rated animated movie named Incredible Family, which is streamed froma remote source and through set-top box 202-4. Set-top box 202-4presents Incredible Family with six advertisements, spaced one at thebeginning of the movie, three in a three-ad block, and two in a two-adblock.

Sensor data is received for all three persons 116 in audience 114; forthis example consider first person 116-1. Assume here that, over thecourse of Incredible Family, that audience-sensing device 104 measures,and then provides at block 402, the following at various times forperson 116-1:

-   -   Time 1, head orientation 3 degrees, no or low-amplitude audio.    -   Time 2, head orientation 24 degrees, no audio.    -   Time 3, skeletal movement (arms), high-amplitude audio.    -   Time 4, skeletal movement (arms and body), high-amplitude audio.    -   Time 5, head movement, facial-feature change (20%),        moderate-amplitude audio.    -   Time 6, detailed facial orientation data, no audio.    -   Time 7, skeletal orientation (missing), no audio.    -   Time 8, facial orientation, respiration rate.

Block 404 determines, based on the sensor data, a state of the personduring the media program. In some cases block 404 determines aprobability for the state or multiple probabilities for multiple states,respectively. For example, block 404 may determine a state likely to becorrect but with less than full certainty (e.g., 90% chance that theperson is laughing). Block 404 may also or instead determine thatmultiple states are possible based on the sensor data, such as a sad orplacid state, and probabilities for each (e.g., sad state 65%, placidstate 35%).

Block 404 may also or instead determine demographics, identity, and/orengagement. Further, methods 400 may skip block 404 and proceed directlyto block 406, as described later below.

In the ongoing example, state module 106 receives the above-listedsensor data and determines the following corresponding states for person116-1:

-   -   Time 1: Looking toward.    -   Time 2: Looking away.    -   Time 3: Clapping.    -   Time 4: Cheering.    -   Time 5: Laughing.    -   Time 6: Smiling.    -   Time 7: Departed.    -   Time 8: Asleep.

At Time 1 state module 106 determines, based on the sensor dataindicating a 3-degree deviation of person 116-1's head from lookingdirectly at the LCD display and a rule indicating that the lookingtoward state applies for deviations of less than 20 degrees (by way ofexample only), that person 116-1's state is looking toward the mediaprogram. Similarly, at Time 2, state module 106 determines person 116-1to be looking away due to the deviation being greater than 20 degrees.

At Time 3, state module 106 determines, based on sensor data indicatingthat person 116-1 has skeletal movement in his arms and audio that ishigh amplitude that person 116-1 is clapping. State module 106 maydifferentiate between clapping and other states, such as cheering, basedon the type of arm movement (not indicated above for brevity).Similarly, at Time 4, state module 106 determines that person 116-1 ischeering due to arm movement and high-amplitude audio attributable toperson 116-1.

At Time 5, state module 106 determines, based on sensor data indicatingthat person 116-1 has head movement, facial-feature changes of 20%, andmoderate-amplitude audio, that person 116-1 is laughing. Various sensordata can be used to differentiate different states, such as screaming,based on the audio being moderate-amplitude rather than high-amplitudeand the facial-feature changes, such as an opening of the mouth and arising of both eyebrows.

For Time 6, audience-sensing device 104 processes raw sensor data toprovide processed sensor data, and in this case facial recognitionprocessing to provide detailed facial orientation data. In conjunctionwith no audio, state module 106 determines that the detailed facialorientation data (here upturned lip corners, amount of eyelids coveringeyes) that person 116-1 is smiling.

At Time 7, state module 106 determines, based on sensor data indicatingthat person 116-1 has skeletal movement moving away from theaudience-sensing device 104, that person 116-1 is departed. The sensordata may indicate this directly as well, such as in cases whereaudience-sensing device 104 does not sense person 116-1's presence,either through no skeletal or head readings or a thermal signature nolonger being received.

At Time 8, state module 106 determines, based on sensor data indicatingthat person 116-1's facial orientation has not changed over a certainperiod (e.g., the person's eyes have not blinked) and a steady, slowrespiration rate that person 116-1 is asleep.

These eight sensor readings are simplified examples for purpose ofexplanation. Sensor data may include extensive data as noted elsewhereherein. Further, sensor data may be received measuring an audience everyfraction of a second, thereby providing detailed data for tens,hundreds, and thousands of periods during presentation of a mediaprogram and from which states or other media reactions may bedetermined.

Returning to methods 400, block 404 may determine demographics,identity, and engagement in addition to a person's state. State module106 may determine or receive sensor data from which to determinedemographics and identity or receive, from audience-sensing device 104,the demographics or identity. Continuing the ongoing example, the sensordata for person 116-1 may indicate that person 116-1 is John Brown, thatperson 116-2 is Lydia Brown, and that person 116-3 is Susan Brown, forexample. Or sensor data may indicate that person 116-1 is six feet, fourinches tall and male (based on skeletal orientation), for example. Thesensor data may be received with or include information indicatingportions of the sensor data attributable separately to each person inthe audience. In this present example, however, assume thataudience-sensing device 104 provides three sets of sensor data, witheach set indicating the identity of the person along with the sensordata.

Also at block 404, the techniques may determine an engagement of anaudience or person in the audience. As noted, this determination can beless refined than that of states of a person, but nonetheless is useful.Assume for the above example, that sensor data is received for person116-2 (Lydia Brown), and that this sensor data includes only head andskeletal orientation:

-   -   Time 1, head orientation 0 degrees, skeletal orientation upper        torso forward of lower torso.    -   Time 2, head orientation 2 degrees, skeletal orientation upper        torso forward of lower torso.    -   Time 3, head orientation 5 degrees, skeletal orientation upper        torso approximately even with lower torso.    -   Time 4, head orientation 2 degrees, skeletal orientation upper        torso back from lower torso.    -   Time 5, head orientation 16 degrees, skeletal orientation upper        torso back from lower torso.    -   Time 6, head orientation 37 degrees, skeletal orientation upper        torso back from lower torso.    -   Time 7, head orientation 5 degrees, skeletal orientation upper        torso forward of lower torso.    -   Time 8, head orientation 1 degree, skeletal orientation upper        torso forward of lower torso.

State module 106 receives this sensor data and determines the followingcorresponding engagement for Lydia Brown:

-   -   Time 1: Engagement High.    -   Time 2: Engagement High.    -   Time 3: Engagement Medium-High.    -   Time 4: Engagement Medium.    -   Time 5: Engagement Medium-Low.    -   Time 6: Engagement Low.    -   Time 7: Engagement High.    -   Time 8: Engagement High.

At Times 1, 2, 7, and 8, state module 106 determines, based on thesensor data indicating a 5-degree-or-less deviation of person 116-2'shead from looking directly at the LCD display and skeletal orientationof upper torso forward of lower torso (indicating that Lydia is leaningforward to the media presentation) that Lydia is highly engaged inIncredible Family at these times.

At Time 3, state module 106 determines that Lydia's engagement level hasfallen due to Lydia no longer leaning forward. At Time 4, state module106 determines that Lydia's engagement has fallen further to mediumbased on Lydia leaning back, even though she is still looking almostdirectly at Incredible Family.

At Times 5 and 6, state module 106 determines Lydia is less engaged,falling to Medium-Low and then Low engagement based on Lydia stillleaning back and looking slightly away (16 degrees) and thensignificantly away (37 degrees), respectively. Note that at Time 7 Lydiaquickly returns to a High engagement, which media creators are likelyinterested in, as it indicates content found to be exciting or otherwisecaptivating.

Methods 400 may proceed directly from block 402 to block 406, or fromblock 404 to block 406 or block 408. If proceeding to block 406 fromblock 404, the techniques determine an interest level based on the typeof media being presented and the person's engagement or state. Ifproceeding to block 406 from block 402, the techniques determine aninterest level based on the type of media being presented and theperson's sensor data, without necessarily first or independentlydetermining the person's engagement or state.

Continuing the above examples for persons 116-1 and 116-2, assume thatblock 406 receives states determined by state module 106 at block 404for person 116-1 (John Brown). Based on the states for John Brown andinformation about the media program, interest module 108 determines aninterest level, either overall or over time, for Incredible Family.Assume here that Incredible Family is both an adventure and a comedyprogram, with portions of the movie marked as having one of these mediatypes. While simplified, assume that Times 1 and 2 are marked as comedy,Times 3 and 4 are marked as adventure, Times 5 and 6 are marked ascomedy, and that Times 7 and 8 are marked as adventure. Revisiting thestates determined by state module 106, consider the following again:

-   -   Time 1: Looking toward.    -   Time 2: Looking away.    -   Time 3: Clapping.    -   Time 4: Cheering.    -   Time 5: Laughing.    -   Time 6: Smiling.    -   Time 7: Departed.    -   Time 8: Asleep.

Based on these states, state module 106 determines for Time 1 that JohnBrown has a medium-low interest in the content at Time 1—if this were ofan adventure or drama type, state module 106 may determine John Brown toinstead be highly interested. Here, however, due to the content beingcomedy and thus intended to elicit laughter or a similar state, interestmodule 108 determines that John Brown has a medium-low interest at Time1. Similarly, for Time 2, interest module 108 determines that John Brownhas a low interest at Time 2 because his state is not only not laughingor smiling but is looking away.

At Times 3 and 4, interest module 108 determines, based on the adventuretype for these times and states of clapping and cheering, that JohnBrown has a high interest level. At time 6, based on the comedy type andJohn Brown smiling, that he has a medium interest at this time.

At Times 7 and 8, interest module 108 determines that John Brown has avery low interest. Here the media type is adventure, though in this caseinterest module 108 would determine John Brown's interest level to bevery low for most types of content.

As can be readily seen, advertisers, media providers, and media creatorscan benefit from knowing a person's interest level. Here assume that theinterest level is provided over time for Incredible Family, along withdemographic information about John Brown. With this information fromnumerous demographically similar persons, a media creator may learn thatmale adults are interested in some of the adventure content but thatmost of the comedy portions are not interesting, at least for thisdemographic group.

Consider, by way of a more-detailed example, FIG. 5, which illustrates atime-based graph 500 having interest levels 502 for forty time periods504 over a portion of a media program. Here assume that the mediaprogram is a movie that includes other media programs—advertisements—attime periods 18 to 30. Interest module 108 determines, as shown, thatthe person begins with a medium interest level, and then bounces betweenmedium and medium-high, high, and very high interest levels to timeperiod 18. During the first advertisement, which covers time periods18-22, interest module 108 determines that the person has a medium lowinterest level. For time periods 23 to 28, however, interest module 108determines that the person has a very low interest level (because he islooking away and talking or left the room, for example). For the lastadvertisement, which covers time period 28 to 32, however, interestmodule 108 determines that the person has a medium interest level fortime periods 29 to 32—most of the advertisement. This can be valuableinformation—the person stayed for the first advertisement, left for themiddle advertisement and the beginning of the last advertisement, andreturned, with medium interest, for most of the last advertisement.Contrast this resolution and accuracy of interest with some conventionalapproaches, which likely would provide no information about how many ofthe people that watched the movie actually watched the advertisements,which ones, and with what amount of interest. If this example is acommon trend with the viewing public, prices for advertisements in themiddle of a block would go down, and other advertisement prices would beadjusted as well. Or, advertisers and media providers might learn toplay shorter advertisement blocks having only two advertisements, forexample. Interest levels 502 also provide valuable information aboutportions of the movie itself, such as through the very high interestlevel at time period 7 and the waning interest at time periods 35-38.

Note that, in some cases, engagement levels, while useful, may be lessuseful or accurate than states and interest levels. For example, statemodule 106 may determine, for just engagement levels, that a person isnot engaged if the person's face is occluded (blocked) and thus notlooking at the media program. If the person's face is blocked by thatperson's hands (skeletal orientation) and audio indicates high-volumeaudio, state module 106, when determining states, may determine theperson to be screaming A screaming state indicates, in conjunction withthe content being horror or suspense, an interest level that is veryhigh. This is but one example of where an interest level can be markedlydifferent from that of an engagement level.

As noted above, methods 400 may proceed directly from block 402 to block406. In such a case, interest module 108, either alone or in conjunctionwith state module 106, determines an interest level based on the type ofmedia (including multiple media types for different portions of a mediaprogram) and the sensor data. By way of example, interest module 108 maydetermine that for sensor data for John Brown at Time 4, which indicatesskeletal movement (arms and body), and high-amplitude audio, and acomedy, athletics, conflict-based talk show, adventure-based video game,tweet, or horror types, that John Brown has a high interest level atTime 4. Conversely, interest module 108 may determine that for the samesensor data at Time 4 for a drama, melodrama, or classical music, thatJohn Brown has a low interest level at Time 4. This can be performedbased on the sensor data without first determining an engagement levelor state, though this may also be performed.

Block 408, either after block 404 or 406, provides the demographics,identity, engagement, state, and/or interest level. State module 106 orinterest module 108 may provide this information to various entities,such as interest module 110, as well as advertisers, media creators, andmedia providers. Providing this information to an advertising entity ormedia provider can be effective to enable the advertising entity tomeasure a value of their advertisements shown during a media program orthe media provider to set advertisement costs. Providing thisinformation to a media creator can be effective to enable the mediacreator to assess a potential value of a similar media program orportion thereof. For example, a media creator, prior to releasing themedia program to the general public, may determine portions of the mediaprogram that are not well received, and thus alter the media program toimprove it.

Providing this information to a rating entity can be effective to enablethe rating entity to automatically rate the media program for theperson. Still other entities, such as a media controller, may use theinformation to improve media control and presentation. A localcontroller may pause the media program responsive to all of the personsin the audience departing the room, for example. Providing mediareactions to interest module 110 are addressed in detail in othermethods described below.

Further, this information may be provided to other entities as well.Providing this information to a rating entity, for example, can beeffective to enable the rating entity to automatically rate the mediaprogram for the person (e.g., four stars out of five or a “thumbs up”).Providing this information to a media controller, for example, mayenable the media controller to improve media control and presentation,such as by pausing the media program responsive to all of the persons inthe audience departing the MOM.

As noted herein, the techniques can determine numerous states for aperson over the course of most media programs, even for 15-secondadvertisements or video snippets. In such a case block 404 is repeated,such as at one-second periods.

Furthermore, state module 106 may determine not only multiple states fora person over time, but also various different states at a particulartime. A person may be both laughing and looking away, for example, bothof which are states that may be determined and provided or used todetermine the person's interest level.

Further still, either or both of state module 106 and interest module108 may determine engagement, states, and/or interest levels based onhistorical data in addition to sensor data or media type. In one case aperson's historical sensor data is used to normalize the person'sengagement, states, or interest levels. If, for example, Susan Brown isviewing a media program and sensor data for her is received, thetechniques may normalize or otherwise learn how best to determineengagement, states, and interest levels for her based on her historicalsensor data. If Susan Brown's historical sensor data indicates that sheis not a particularly expressive or vocal person, the techniques mayadjust for this history. Thus, lower-amplitude audio may be sufficientto determine that Susan Brown laughed compared to an amplitude of audioused to determine that a typical person laughed.

In another case, historical engagement, states, or interest levels ofthe person for which sensor data is received are compared withhistorical engagement, states, or interest levels for other people.Thus, a lower interest level may be determined for Lydia Brown based ondata indicating that she exhibits a high interest for almost every mediaprogram she watches compared to other people's interest levels (eithergenerally or for the same media program). In either of these cases thetechniques learn over time, and thereby can normalize engagement,states, and/or interest levels.

FIG. 6 depicts methods 600 enabling a user interface presenting mediareactions. Methods 600 are responsive to determining one or more personsto which a selected media program has been presented, though this is notrequired.

People often want to know how other people reacted to a particularprogram, such as to compare his or her own reactions to theirs or todecide whether or not to watch the program. By way of example, consideragain the above example of Melody Pond wishing to determine which of herfriends and family watched a particular episode of The Office and theirreactions to it. Here assume that Melody does not yet know which of herfriends and family have seen the episode and that she has not yet seenthe episode.

Block 602 receives selection of a media program. This can be throughselecting an icon, graphic, or label for the media program, or receivingterms for a search inquiry by which to determine the media program, orother manners. Assume here that Melody enters, into a data entry field(not shown) of user interface 112, the search terms: “the office”. Inresponse, interface module 110 presents likely candidates, such as amovie named “the office” and the last three episodes of the televisionshow The Office. In response, Melody selects the most-recent episode ofthe television show, episode 104.

Block 604 determines a person to which a selected media program has beenpresented and passive sensor data has been sensed during thepresentation of the selected media program to the person or for whichmedia reactions have been received. Block 604 may determine persons froma group that have indicated a selection to share media reactions withthe user of the user interface, such as friends and family. Block 604may also or instead find other people that the user doesn't know, suchas published critics or famous people for which passive sensor data hasbeen recorded. For this example, Melody's friends and family are found,namely Calvin Pond, Lydia Brown, and Amelia Pond.

At some point media reactions for a person are received by interfacemodule 110, whether from a local or remote state module 106 or interestmodule 108 (e.g., at computing device 202 or remote device 302,respectively).

Block 606 presents, in a user interface, a media reaction for theperson, the media reaction determined based on the passive sensor datasensed during the presentation of the selected media program to theperson. The media reaction can include any of the reactions noted above,such as an overall rating for the program or a time-based graph showingreactions over the course of the program, to name just two.

A time-based graph shows media reactions for the person over at leastsome portion of the media program, such as at particular time periods.Interface module 110 may present the media reactions in alower-resolution format, such as by aggregating or averaging these mediareactions. If media reactions are received in one-second time periods,for example, interface module 110 may average these into larger (e.g.,five- or ten-second) time periods. By so doing, a user may more-easilyconsume the information. Interface module 110, however, may also enablethe user to select to view the original, higher resolution as well.

Consider, for example, FIG. 7, which illustrates tablet computer 202-2with a graphical user interface 700 presenting time-based graphs andratings for three persons for a media program. Continuing the ongoingexample, three persons are determined: Amelia Pond, Calvin Pond, andLydia Brown. For breadth of illustration, each of the time-based graphsshow different categories of media reactions, namely engagement,interest level, and state, each of which are described above, as well asan average of these three in one time-based graph. The four graphs eachpresent 44 media reactions (separately or as an average), one for each30-second time period of 22 total minutes of media content.

Interest level graph 702 shows interest levels over the 44 time periodsfor Calvin Pond. Interest level graph 702 shows that Calvin had a mediumto very low interest level through much of the program.

Engagement graph 704 shows engagements over the 44 time periods forLydia Brown. Engagement graph 704 shows that Lydia was highly engagedfor much of the program. Here assume that Lydia watched the programthrough a mobile device having a lower-resolution audience-sensingdevice (e.g., a camera in smart phone 202-3) and thus engagements forLydia were received.

State graph 706 shows states over the 44 time periods for Amelia Pond.State graph 706 shows that Amelia was laughing or smiling through mostof the program. For visual brevity, state graph 706 shows four states,laughing, smiling, looking-away, and crying.

Average reaction graph 708 shows an average of Calvin's interest levels,Lydia's engagement, and Amelia's states. This can be useful to the userto gauge how his or her friends (in this case Melody Pond) liked themedia program. An average of reactions can be any mix of reactions orjust include states, engagements, or interest levels of multiple people,though this is not required. Here the average includes all three ofinterest levels, engagements, and states. To produce this average, thetechniques may convert any non-numerical reactions, such as Amelia'sstates, to a number for averaging, as well as normalize and/or weightone or more of the reactions.

Graphical user interface 700 also presents four ratings, 710, 712, 714,and 716. These ratings are based on the reactions received or can bereceived themselves. Thus, these ratings can be determined automaticallyby state module 106, interest module 108, or interface module 110, orthey may be based on a manual selection by the person. The first rating,710, indicates that Calvin's rating is 2.8 out of five, whichcorresponds to interest level graph 702. The second rating, 712,indicates that Lydia's rating is 4.3 out of five, which corresponds toher high engagement throughout the program. The third rating, 714,summarizes Amelia's most-common state, that of laughing, and presents alaughing graphic to show this rating. The fourth rating, 716, is anaverage of the first, second, and third ratings, with the third ratingconverted into a number (4.5) to produce the fourth rating of 3.9.

Graphical user interface 700 presents an identity of the media program(The Office, Episode 104) at 718 and identifiers for individual personsat 720, 722, and 724, and a conglomerate of these individual persons at726, which here are the person's names, though icons, avatars, graphics,or other identifiers may be used.

Returning to methods 600, the techniques may optionally enableselection, through the user interface, to display the media program or aportion thereof. This option is shown at block 608. Interface module 110can enable selection to play all of the media program, for example,through selection of the identifier (here 718 in graphical userinterface 700). A portion or portions may also be selected. Consider acase where the media reactions are presented with some correlation orassociated with portions of the media program, such as with time periodsin the above-described time-based graphs.

In such a case, block 608 enables selection, through the time-basedgraph of the user interface, to display a portion of the media programcorresponding to a media reaction, such as a very high engagement inengagement graph 704 or a crying state in state graph 706. Thus, a userof graphical user interface 700, such as Melody Pond, may select acrying icon 728 in state graph 706 to see the portion of The Officeduring which Amelia cried. Further, interface module 110 enables usersto select to see the media program in conjunction with a representationof a person's media reactions, such as through an animated avatarperforming a state (e.g., laughing, clapping, or cheering) or indicatingan engagement or interest level of the media reactions.

Likewise, consider a second case where a portion or portions of themedia program may be selected through graphical user interface 700.Assume that a user wishes to see the best parts of a media program—suchas parts where the user or one or more friends laughed at a comedy orcried during a melodrama. In the ongoing example, assume Melody Pondwishes to see the funniest parts of The Office, Episode 104, based onall three of her friend's reactions. In such a case, block 608 enablesselection to display a portion or portions of the media program. HereMelody Pond may select a “Best Part!” control 730 or a “Best Five!”control 732 to see portions of The Office where all of her friendslaughed (control 730) or all of them laughed or at least smiled (control732). As in the above examples, interface module 110 may enable users tosee these portions alone or in conjunction with a representation of oneor more persons' media reactions.

Block 610, responsive to selection of the media program or portionthereof, causes the media program or portion to be presented. Interfacemodule 110, for example, may cause a television to present the portionof the media program or pop up a window showing the portion over orwithin graphical user interface 700. In cases where the media program isselected to be presented in conjunction with a representation of theperson's media reactions, interface module 110 may render the programwith an avatar presented in a corner of the display, for example.Interface module 110 is not limited to representing reactions of oneperson, interface module 110 may present The Office with avatars for allthree of Calvin, Lydia, and Amelia to show their reactions during theprogram.

Concluding the ongoing example, interface module 110 at block 610, inresponse to selection of crying icon 722, presents a 30-second clip ofThe Office within a portion of graphical user interface 700.

FIG. 8 depicts methods 800 enabling a user interface presenting mediareactions. Methods 800 present media reactions for media programsresponsive to determining which media programs a selected person hasseen, though this is not required. The techniques may perform methods400, 600, and/or 800, alone or in conjunction with each other, in wholeor in part.

Block 802 receives selection of a person. The person may be selectedthrough a user interface through which media reactions will bedisplayed, such as through a graphical representation or otheridentifier of the person, though this is not required.

Block 804 determines a media program that has been presented to theselected person and for which passive sensor data has been sensed duringthe presentation of the media program to the selected person.

By way of example, consider a case where a user of user interface 112 ofFIG. 1 wishes to know what his best friend has watched over the lastmonth. Assume that the user knows that he has similar tastes as his bestfriend and so would like to find out which programs his best friendliked. The techniques can enable the user to quickly and easily see whathis best friend has watched as well as overall ratings for theseprograms and/or reactions to various portions of the programs.

Block 806 presents, in a user interface, a time-based graph for theperson, the time-based graph showing media reactions for the person anddetermined based on the passive sensor data sensed during thepresentation of the selected media program to the person. Interfacemodule 110 may repeat block 804 and 806 to show numerous graphs fornumerous media programs, such as time-based graphs for three movies,nine television shows, four music videos, twelve tweets, and six videoclips. Interface module 110 may arrange these by media category, such asmovies together, then television shows, and so forth, or by rating(e.g., highest rating to lowest rating), or by order viewed, to namejust a few.

The preceding discussion describes methods relating to determining orpresenting a person's media reactions based on passive sensor data.Aspects of these methods may be implemented in hardware (e.g., fixedlogic circuitry), firmware, software, manual processing, or anycombination thereof. A software implementation represents program codethat performs specified tasks when executed by a computer processor. Theexample methods may be described in the general context ofcomputer-executable instructions, which can include software,applications, routines, programs, objects, components, data structures,procedures, modules, functions, and the like. The program code can bestored in one or more computer-readable memory devices, both localand/or remote to a computer processor. The methods may also be practicedin a distributed computing mode by multiple computing devices. Further,the features described herein are platform-independent and can beimplemented on a variety of computing platforms having a variety ofprocessors.

These techniques may be embodied on one or more of the entities shown inFIGS. 1-3 and 9 (device 900 is described below), which may be furtherdivided, combined, and so on. Thus, these figures illustrate some ofmany possible systems or apparatuses capable of employing the describedtechniques. The entities of these figures generally represent software,firmware, hardware, whole devices or networks, or a combination thereof.In the case of a software implementation, for instance, the entities(e.g., state module 106, interest module 108, and interface module 110)represent program code that performs specified tasks when executed on aprocessor (e.g., processor(s) 204 and/or 306). The program code can bestored in one or more computer-readable memory devices, such as media206 and/or 308 or computer-readable media 914 of FIG. 9.

Example Device

FIG. 9 illustrates various components of example device 900 that can beimplemented as any type of client, server, and/or computing device asdescribed with reference to the previous FIGS. 1-8 to implementtechniques enabling a user interface presenting a media reaction. Inembodiments, device 900 can be implemented as one or a combination of awired and/or wireless device, as a form of television mobile computingdevice (e.g., television set-top box, digital video recorder (DVR),etc.), consumer device, computer device, server device, portablecomputer device, user device, communication device, video processingand/or rendering device, appliance device, gaming device, electronicdevice, System-on-Chip (SoC), and/or as another type of device orportion thereof. Device 900 may also be associated with a user (e.g., aperson) and/or an entity that operates the device such that a devicedescribes logical devices that include users, software, firmware, and/ora combination of devices.

Device 900 includes communication devices 902 that enable wired and/orwireless communication of device data 904 (e.g., received data, datathat is being received, data scheduled for broadcast, data packets ofthe data, etc.). Device data 904 or other device content can includeconfiguration settings of the device, media content stored on the device(e.g., media programs 210), and/or information associated with a user ofthe device. Media content stored on device 900 can include any type ofaudio, video, and/or image data. Device 900 includes one or more datainputs 906 via which any type of data, media content, and/or inputs canbe received, such as human utterances, user-selectable inputs, messages,music, television media content, recorded video content, and any othertype of audio, video, and/or image data received from any content and/ordata source.

Device 900 also includes communication interfaces 908, which can beimplemented as any one or more of a serial and/or parallel interface, awireless interface, any type of network interface, a modem, and as anyother type of communication interface. Communication interfaces 908provide a connection and/or communication links between device 900 and acommunication network by which other electronic, computing, andcommunication devices communicate data with device 900.

Device 900 includes one or more processors 910 (e.g., any ofmicroprocessors, controllers, and the like), which process variouscomputer-executable instructions to control the operation of device 900and to enable techniques enabling a user interface presenting a mediareaction. Alternatively or in addition, device 900 can be implementedwith any one or combination of hardware, firmware, or fixed logiccircuitry that is implemented in connection with processing and controlcircuits which are generally identified at 912. Although not shown,device 900 can include a system bus or data transfer system that couplesthe various components within the device. A system bus can include anyone or combination of different bus structures, such as a memory bus ormemory controller, a peripheral bus, a universal serial bus, and/or aprocessor or local bus that utilizes any of a variety of busarchitectures.

Device 900 also includes computer-readable storage media 914, such asone or more memory devices that enable persistent and/or non-transitorydata storage (i.e., in contrast to mere signal transmission), examplesof which include random access memory (RAM), non-volatile memory (e.g.,any one or more of a read-only memory (ROM), flash memory, EPROM,EEPROM, etc.), and a disk storage device. A disk storage device may beimplemented as any type of magnetic or optical storage device, such as ahard disk drive, a recordable and/or rewriteable compact disc (CD), anytype of a digital versatile disc (DVD), and the like. Device 900 canalso include a mass storage media device 916.

Computer-readable storage media 914 provides data storage mechanisms tostore device data 904, as well as various device applications 918 andany other types of information and/or data related to operationalaspects of device 900. For example, an operating system 920 can bemaintained as a computer application with computer-readable storagemedia 914 and executed on processors 910. Device applications 918 mayinclude a device manager, such as any form of a control application,software application, signal-processing and control module, code that isnative to a particular device, a hardware abstraction layer for aparticular device, and so on.

Device applications 918 also include any system components, engines, ormodules to implement techniques enabling a user interface presenting amedia reaction. In this example, device applications 918 can includestate module 106, interest module 108, or interface module 110.

CONCLUSION

Although embodiments of techniques and apparatuses enabling a userinterface presenting a media reaction have been described in languagespecific to features and/or methods, it is to be understood that thesubject of the appended claims is not necessarily limited to thespecific features or methods described. Rather, the specific featuresand methods are disclosed as example implementations enabling a userinterface presenting a media reaction.

1. A computer-implemented method comprising: determining a person towhich a selected media program has been presented and passive sensordata has been sensed during the presentation of the selected mediaprogram to the person; and presenting, in a user interface, an animatedavatar preforming a media reaction for the person, the media reactionincluding a clapping, cheering, laughing, screaming, looking toward,looking away, smiling, departed, crying, or sleeping state, the mediareaction determined based on the passive sensor data sensed during thepresentation of the selected media program to the person.
 2. Acomputer-implemented method as described in claim 1, further comprisingpresenting multiple other animated avatars performing multiple othermedia reactions in a time-based state graph for the person. 3.(canceled)
 4. A computer-implemented method as described in claim 1,further comprising presenting a rating automatically determined for theperson based on the passive sensor data.
 5. A computer-implementedmethod as described in claim 1, wherein determining the persondetermines the person from a group of persons that have selected toshare media reactions with a second person, the second person associatedwith the user interface or associated with a computing device throughwhich the user interface is presented.
 6. A computer-implemented methodas described in claim 1, further comprising determining one or moreother persons to which the selected media program has been presented andother passive sensor data has been sensed during the presentation of theselected media program to the one or more other persons and presenting,in the user interface, another media reaction for each of the one ormore other persons.
 7. A computer-implemented method as described inclaim 1, further comprising enabling selection, through the userinterface, to display the selected media program or a portion thereof,and responsive to selection, causing display of the selected mediaprogram or portion thereof.
 8. A computer-implemented method asdescribed in claim 7, wherein causing display of the selected mediaprogram or portion thereof presents the selected media program orportion thereof in the user interface.
 9. A computer-implemented methodas described in claim 7, further comprising causing a representation ofthe media reaction to be displayed with display of the selected mediaprogram or portion thereof.
 10. A computer-implemented method asdescribed in claim 1, further comprising, prior to presenting theanimated avatar performing the media reaction for the person,determining the media reaction based on the passive sensor data sensedduring the presentation of the selected media program to the person. 11.A computer-implemented method as described in claim 1, furthercomprising, prior to presenting the animated avatar performing the mediareaction for the person, receiving a search inquiry through the userinterface and determining, based on the search inquiry, the selectedmedia program.
 12. A computer-implemented method comprising: determininga media program that has been presented to a selected person and forwhich passive sensor data has been sensed during the presentation of themedia program to the selected person; and presenting, in a userinterface, an animated avatar performing a media reaction for theperson, the media reaction including a clapping, cheering, laughing,screaming, looking toward, looking away, smiling, departed, crying, orsleeping state, the media reaction determined based on the passivesensor data sensed during the presentation of the selected media programto the person.
 13. A computer-implemented method as described in claim12, further comprising presenting multiple other animated avatarsperforming multiple other media reactions in a time-based state graph.14. A computer-implemented method as described in claim 12, furthercomprising determining multiple other media programs that have beenpresented to the selected person and for which passive sensor data hasbeen sensed during the presentations of the media programs and,responsive to determining the multiple other media programs, presentinganother media reaction for each of the multiple other media programswithin the user interface.
 15. A computer-implemented method asdescribed in claim 14, wherein the other media reactions each includemultiple other media reactions and presenting another media reaction foreach of the multiple other media programs within the user interfacepresents time-based graphs for each of the other media reactions.
 16. Acomputer-implemented method as described in claim 15, wherein presentingthe time-based graphs presents the time-based graphs in the userinterface arranged by order of viewing, media category, or rating forthe respective media programs.
 17. A computer-implemented methodcomprising: receiving media reactions for a person, the media reactionsdetermined based on sensor data passively sensed at time periods duringwhich a media program is presented to the person; presenting, in a userinterface, a time-based graph, the time-based graph showing the mediareactions for the person at the time periods; enabling selection,through the time-based graph of the user interface, to display a portionof the media program associated with one of the media reactions; andresponsive to selection of the portion of the media program, causing:the portion of the media program to be presented; and an animated avatarto be presented in conjunction with the portion of the media program,the animated avatar clapping, cheering, laughing, screaming, lookingtoward, looking away, smiling, departed, crying, or sleeping incorrespondence with one of the media reactions.
 18. (canceled) 19.(canceled)
 20. A computer-implemented method as described in claim 17,wherein the media reactions include states of the person, the statesincluding one or more of: a sad, a related talking, an unrelatedtalking, a disgusted, an afraid, a scowling, a placid, a surprised, anangry, a waving, a leaning-away, or a leaning-toward state.
 21. Acomputer-implemented method as described in claim 12, further comprisingpresenting a rating automatically determined for the person based on thepassive sensor data.
 22. A computer-implemented method as described inclaim 12, further comprising determining the media reaction.
 23. Acomputer-implemented method as described in claim 12, wherein the mediareaction further includes a sad, a related talking, an unrelatedtalking, a disgusted, an afraid, a scowling, a placid, a surprised, awaving, a leaning-away, a leaning-toward, or an angry state.